Systems and methods for player input motion compensation by anticipating motion vectors and/or caching repetitive motion vectors

ABSTRACT

Systems and methods for reducing latency through motion estimation and compensation techniques are disclosed. The systems and methods include a client device that uses transmitted lookup tables from a remote server to match user input to motion vectors, and tag and sum those motion vectors. When a remote server transmits encoded video frames to the client, the client decodes those video frames and applies the summed motion vectors to the decoded frames to estimate motion in those frames. In certain embodiments, the systems and methods generate motion vectors at a server based on predetermined criteria and transmit the generated motion vectors and one or more invalidators to a client, which caches those motion vectors and invalidators. The server instructs the client to receive input from a user, and use that input to match to cached motion vectors or invalidators. Based on that comparison, the client then applies the matched motion vectors or invalidators to effect motion compensation in a graphic interface. In other embodiments, the systems and methods cache repetitive motion vectors at a server, which transmits a previously generated motion vector library to a client. The client stores the motion vector library, and monitors for user input data. The server instructs the client to calculate a motion estimate from the input data and instructs the client to update the stored motion vector library based on the input data, so that the client applies the stored motion vector library to initiate motion in a graphic interface prior to receiving actual motion vector data from the server. In this manner, latency in video data streams is reduced.

RELATED APPLICATIONS

This application claims the benefit of the following U.S. ProvisionalApplication No. 62/488,526, filed Apr. 21, 2017, No. 62/634,464, filedFeb. 23, 2018, No. 62/640,945, filed Mar. 9, 2018, and No. 62/644,164,filed Mar. 16, 2018.

BACKGROUND OF THE INVENTION

Remote gaming applications, in which a server-side game is controlled bya client-side player, have attempted to encode the video output from athree-dimensional (3D) graphics engine in real-time using existing orcustomized encoders. However, the interactive nature of video games,particularly the player feedback loop between video output and playerinput, makes game video streaming much more sensitive to latency thantraditional video streaming. Existing video coding methods can tradecomputational power, and little else, for reductions in encoding time.New methods for integrating the encoding process into the videorendering process can provide significant reductions in encoding timewhile also reducing computational power, improving the quality of theencoded video, and retaining the original bitstream data format topreserve interoperability of existing hardware devices.

Unlike regular video playback, video games have a unique player input tovideo feedback loop. Players are very sensitive to latency between inputand video output. High latency in this player input-feedback loop hasbeen a significant stumbling block for video game streaming applicationsin which a server-hosted video game instance is controlled by a remoteplayerAny process that can reduce the time between input and feedbackwill directly improve user experience.

The client hardware in a game streaming environment may have varyinglevels of computational performance, but dedicated H.264 hardwaredecoders are becoming more ubiquitous, even in mobile and otherlow-power devices. A hardware decoder is good at performing a smallselection of computations, such as motion compensation, which areregularly performed according to the H.264 coding standard. Thestrengths of the dedicated decoding hardware can be exploited to providea better player experience in a game streaming environment regardless ofthe overall computational power of the client.

In local, non-streamed rendering applications, the game engine can addseveral frames of latency between player input and video feedback. Ingame streaming applications, additional latency is introduced to theplayer input-feedback cycle because player input must travel through thenetwork to the remote server and video output must be encoded,transmitted, and decoded before the player receives feedback. For someplayer inputs, the client can estimate the results on video feedback byperforming motion compensation immediately, cutting out the networklatency.

Player input motion compensation is, at its most basic, a technique ofshifting groups of pixels in order to sacrifice some image accuracy fora decrease in input-feedback latency in situations where a video game isrunning on a server while being controlled remotely by a networkedclient. This technique is good at reducing player-feedback latency forinputs that result in consistent motion vectors, such as player viewrotations in first-person games.

In a video game, player-context is defined as the current game statewhich is the result of previous player actions, inputs, and decisions.The client in a game streaming system is naïve to the player-context,that is, the client receives only the video output and none of thegame-state information that will determine the results of certain playerinputs. There are a wide range of inputs that result in unique yetpredictable motion outcomes based on game-state information. Theseinputs would benefit from a reduction in player-feedback latency butcannot be pre-cached on the client for traditional player input motioncompensation because the client will not have player-contextinformation. Additionally, the player-context permutation space may betoo exhaustive to pre-generate motion vectors for methods such as cachedrepetitive motion vectors. These systems and methods are described inU.S. Provisional Application Nos. 62/488,526; 62/634,464; and62/640,945; all three of which are incorporated here in theirentireties. The game-server can compensate by generating anticipatedmotion vectors and updating the client's input motion compensation cacheas the player-context changes. This allows the client to use playerinput motion compensation techniques for a limited set ofcontext-dependent inputs, resulting in input-feedback latency reduction.

U.S. Pat. No. 9,661,351 (“the '351 Patent”), discloses systems andmethods for skipping a frame during transmission from a server to aclient device, where, in response to detecting the skipped frame, theclient device generating a predicted frame that replaces the skippedframe in the compressed video stream, the predicted frame beinggenerated based on extending delta information from one or more previousframes decoded by the client device. For this technology, theclient-side frame prediction of one or more reconstructed or predictedframes is used following a skipped frame based on the data (e.g., motionvectors, residuals, etc.) of one or more preceding frames. Thetechnology also prioritizes bit allocation and/or subfeature encoding.Encoded Network Abstraction Layer Units (NALUs) could be split into (1)motion vectors and (2) residuals. Instead of actually skipping a frame,the apparatus may just send minimal encoding data as prioritized. Forexample it could send just motion vectors if motion is prioritized. Thepresent invention is superior to the technology of the '351 Patent atleast because the '351 Patent does not disclose a client device thatuses transmitted lookup tables from a server to match user input tomotion vectors and tags and sums those motion vectors. The '351 Patentalso does not disclose the application of those summed motion vectors tothe decoded frames to estimate motion in those frames. The presentinvention is also superior because it reduces input-feedback latency,which is significantly reduced by using player input motion compensationinstead of waiting for the server to return output video.

U.S. Pat. No. 8,678,929, (“the '929 Patent), is directed to theoperation of a networked, interactive game system. The disclosed methodsare geared towards reducing network lag by determining course-correctedpositional values for a shared global object through two-way blending.The “blending” discussed in the patent includes the steps of the networksimulation sending the calculation of a pose for the local player on thelocal console. The console blends the local and network simulations. Themethods also blend shared global objects by using the blended pose ofthe local player to determine a positional value for the shared globalobject on the local console. The present invention is again superior tothe technology of the '929 Patent at least because the '929 Patent doesnot disclose a client device that uses transmitted lookup tables from aserver to match user input to motion vectors and tags and sums thosemotion vectors. The '929 Patent also does not disclose the applicationof those summed motion vectors to the decoded frames to estimate motionin those frames. The present invention is also superior because it doesnot require the presence of a client with extremely high processingpower, reduces input-feedback latency, which is significantly reduced byusing player input motion compensation instead of waiting for the serverto return output video.

U.S. Pat. No. 8,069,258, (“the '258 Patent”) is directed to using localframe processing to reduce apparent network lag of multiplayersimulations. The methods described include intercepting inputs from alocal user, determining the state data of a remote object in a networksimulation from a previous game frame, and determining the interactionsof nondeterministic objects from multiple game systems that are part ofthe networked simulation. That interaction data, along with the statedata and local input, is used to generate a local simulation of thevideo frame. In this manner, the local and network simulations can runasynchronously for a single frame, with each frame corresponding to asingle time phase within the game. This allows the local simulationupdates in real-time during networked gameplay, while remaining(essentially) synchronized with the network. The present invention isonce more superior to the technology of the '258 Patent at least becausethe '258 Patent does not disclose a client device that uses transmittedlookup tables from a server to match user input to motion vectors andtags and sums those motion vectors. The '929 Patent also does notdisclose the application of those summed motion vectors to the decodedframes to estimate motion in those frames. The present invention is alsosuperior because it does not require the presence of a client withextremely high processing power, reduces input-feedback latency, whichis significantly reduced by using player input motion compensationinstead of waiting for the server to return output video.

U.S. Pat. No. 9,665,334 B2 (“the '334 Patent”), discloses systems andmethods for rendering protocols applying multiple processes and acompositor to render the combined graphics on a display. The technologyoperates as follows: when the server simultaneously provides a gamescreen to a number of client devices, the calculation load of therendering processing on the server becomes heavy in, for example, a gamecontent which requires quick responsiveness. That is, the number ofclient devices to which the server can provide a screen is limiteddepending on its rendering performance and required responsiveness. Bycontrast, when each client device is controlled to execute processingwhich can be executed by general rendering performance to share therendering processes between the server and client device, a screen canbe provided to more client devices. Also, in general, a game screenwhich is rendered without applying texture mapping has high compressionefficiency, and can be sent with a smaller bandwidth via a network suchas the Internet. The present invention is superior to the technologydiscussed in the '334 Patent at least because it does not disclosegenerating motion vectors at a server based on predetermined criteriaand transmitting the generated motion vectors and one or moreinvalidators to a client, which caches those motion vectors andinvalidators. It further does not disclose having the server instructthe client to receive input from a user, and use that input to match tocached motion vectors or invalidators, where those vectors orinvalidators are used in motion compensation. The present invention isalso superior because it reduces input-feedback latency, which issignificantly reduced by using player input motion compensation insteadof waiting for the server to return output video.

U.S. Pat. No. 9,736,454 (“the '454 Patent”), discloses systems andmethods for encoding comprising examining availability of a depth blockco-located with a texture block, determining a prediction method for atexture block on the basis of availability of a co-located depth block,and deriving a first prediction block for the texture block on the basisof the availability of the co-located depth block. Again, the presentinvention is superior to the technology discussed in the '454 Patent atleast because it does not disclose generating motion vectors at a serverbased on predetermined criteria and transmitting the generated motionvectors and one or more invalidators to a client, which caches thosemotion vectors and invalidators. It further does not disclose having theserver instruct the client to receive input from a user, and use thatinput to match to cached motion vectors or invalidators, where thosevectors or invalidators are used in motion compensation. The presentinvention is also superior because it reduces input-feedback latency,which is significantly reduced by using player input motion compensationinstead of waiting for the server to return output video.

U.S. Pat. No. 9,705,526 (“the '526 Patent”), discloses systems andmethods for entropy encoding in media and image applications. Thetechnology discloses a system where compression begins with receiving asource of image and or video data as indicated. A lossless compressionscheme is then applied. A predictor/delta computation unit then takesthe input and tries to reduce the redundancy in the input data using adelta computation between neighboring input elements. Then these valuesare encoded using a predefined statistical modeling in an entropyencoder to produce the compressed image and/or video data. Similar tothe above, the present invention is superior to the technology discussedin the '526 Patent at least because it does not disclose generatingmotion vectors at a server based on predetermined criteria andtransmitting the generated motion vectors and one or more invalidatorsto a client, which caches those motion vectors and invalidators. Itfurther does not disclose having the server instruct the client toreceive input from a user, and use that input to match to cached motionvectors or invalidators, where those vectors or invalidators are used inmotion compensation. The present invention is also superior because itreduces input-feedback latency, which is significantly reduced by usingplayer input motion compensation instead of waiting for the server toreturn output video.

U.S. Pat. No. 8,873,636 B2 (“the '636 Patent”), is directed to amoving-image distribution server (such as one running an online game)that provides coded image data to the user's PC, running the localinstance of the game. In order to perform this process, in relevantdetail, the CPU of the of the user's PC specifies the region to bereferred to in order to decode the motion vector associated with theselected block in the preceding frame screen. It does so by referring tothe motion vector associated with the selected block (a vector that isincluded in the preprocessing information it is provided) and extractsthe image of the region as a reference image. As is the case with theother references, the present invention is superior to the technologydiscussed in the '636 Patent at least because it does not disclosegenerating motion vectors at a server based on predetermined criteriaand transmitting the generated motion vectors and one or moreinvalidators to a client, which caches those motion vectors andinvalidators. It further does not disclose having the server instructthe client to receive input from a user, and use that input to match tocached motion vectors or invalidators, where those vectors orinvalidators are used in motion compensation. The present invention isalso superior because it reduces input-feedback latency, which issignificantly reduced by using player input motion compensation insteadof waiting for the server to return output video.

International Patent Publication No. WO2009138878 A2 (“the '878Publication”) is directed to processing and streaming multipleinteractive applications in a centralized streaming application server,while controlling the levels of detail and post-filtering of variousrendered objects. In the system, a centralized interactive applicationserver, at its video pre-processor, performs spatial and temporalfiltering on the frame sequence prior to encoding a compressed stream ofaudio-visual content to client devices, which decode the compressedstream and display the content. The GPU command processor of thecentralized interactive application server includes a module that alsocomputes a motion compensation estimate for each macroblock in thetarget encoded frame in the video encoder. Nevertheless, the presentinvention remains superior to the technology discussed in the '878Publication at least because it does not disclose generating motionvectors at a server based on predetermined criteria and transmitting thegenerated motion vectors and one or more invalidators to a client, whichcaches those motion vectors and invalidators. It further does notdisclose having the server instruct the client to receive input from auser, and use that input to match to cached motion vectors orinvalidators, where those vectors or invalidators are used in motioncompensation. The present invention is also superior because it reducesinput-feedback latency, which is significantly reduced by using playerinput motion compensation instead of waiting for the server to returnoutput video.

U.S. Pat. No. 9,358,466 B2 (“the '466 Patent”), is directed to improvingvideogame performance through the reuse of cached data. The systemsdisclosed score cache performance of different generated video gamemissions at least in part by identifying the digital assets used, anddetermining whether the identified digital assets are in a cache. Cachescores can be calculated based on a cache re-use rate corresponding to aproportion of digital assets for a mission that are already in a cache.Other techniques for generating cache scores may account for factorssuch as the overall size of combined digital assets for a mission thatare already in a cache and/or overall size of combined digital assetsfor a mission that are not already in a cache. By caching data in thismanner, that data, and other non-cached data requests, become moreefficient. The present invention remains superior to the technologydiscussed in the '466 Patent at least because it does not disclosecaching repetitive motion vectors, calculating a motion estimate fromthe input data or updating the stored motion vector library based on theinput data, so that a client can use the stored motion vector library toinitiate motion prior to receiving actual motion vector data from aserver.

U.S. Pat. No. 6,903,662 B2 (“the '662 Patent), is directed to aconfigurable computer input device, specifically one that caches inputdata to maintain quicker response times. The system operates by checkingkey presses against an internal memory (or cache) to determine if theparticular key has been previously identified. If the system has notcommunicated with the key before this point, the system may thenretrieve data corresponding to the input from its key memory. The systemwill then update its internal memory (cache) with the key identity andits corresponding data. The system may then send the input data from thekey to the host computer. However, the next time the system encountersthe same key in the pressed state, it can query its own memory (cache)instead of retrieving the same scan code from the key memory again. Thepresent invention is superior to the technology discussed in the '662Patent at least because it does not disclose caching repetitive motionvectors, calculating a motion estimate from the input data or updatingthe stored motion vector library based on the input data, so that aclient can use the stored motion vector library to initiate motion priorto receiving actual motion vector data from a server.

Japanese Patent No. JP6129865B2 (“the '865 Patent”), discloses systemsand methods for transmit[ting] the rendered game content data for thesubsets of path segments to a game client for caching on the local cacheso that the game content data may be available when needed duringreal-time game play. Again, the present invention is superior to thetechnology discussed in the '865 Patent at least because it does notdisclose caching repetitive motion vectors, calculating a motionestimate from the input data or updating the stored motion vectorlibrary based on the input data, so that a client can use the storedmotion vector library to initiate motion prior to receiving actualmotion vector data from a server.

U.S. Pat. No. 9,762,919 (“the '919 Patent”), discloses systems andmethods for caching reference data in a block processing pipeline. The'919 Patent technology discloses a cache memory (e.g., a fullyassociative cache) that may be implemented, for example in a local (tothe pipeline) memory such as SRAM (static random access memory), towhich portions of the chroma reference data (e.g., 64-byte memoryblocks) corresponding to motion vectors determined for macroblocks atearlier stages of the pipeline may be prefetched from the memory. Chromacache logic may maintain the cache, and may extend over multiple stagesof the pipeline. Fetches for the motion vectors of a given macroblockpassing through the pipeline may be initiated by the chroma cache logicone or more stages prior to the chroma motion compensation stage toprovide time (i.e., multiple pipeline cycles) to read the respectivememory blocks from the memory into the cache before chroma motioncompensation needs it. However, the '919 Patent remains deficientcompared to the present invention. The present invention is superior tothe technology discussed in the '919 Patent at least because it does notdisclose caching repetitive motion vectors, calculating a motionestimate from the input data or updating the stored motion vectorlibrary based on the input data, so that a client can use the storedmotion vector library to initiate motion prior to receiving actualmotion vector data from a server.

As is apparent from the above discussion of the state of the art in thistechnology, there is a need in the art for an improvement to the presentcomputer technology related to the encoding of real-time gameenvironments.

SUMMARY OF THE INVENTION

It is therefore an object of the present invention to disclose systemsand methods for reducing latency through motion compensation techniquesin which a client device uses transmitted lookup tables from a server tomatch user input to motion vectors, and tag and sum those motionvectors. When a remote server transmits encoded video frames to theclient, the client decodes those video frames and applies the summedmotion vectors to the decoded frames to estimate motion in those frames.

It is another object of the present invention to disclose systems andmethods in which the encoded video frames are decoded without residualhandling.

It is yet another object of the present invention to disclose systemsand methods for reducing latency through motion compensation techniquesin which the client applies one or more smoothing functions to thesummed tagged motion vectors in the queue.

It is yet another object of the present invention to disclose systemsand methods for reducing latency through motion compensation techniquesin which the tags associated with the motion vectors at the clientdevice are chronological in nature.

It is another object of the invention to provide systems and methods forreducing input-feedback latency by generating motion vectors at a serverbased on predetermined criteria and transmitting the generated motionvectors and one or more invalidators to a client, which caches thosemotion vectors and invalidators. The server instructs the client toreceive input from a user, and use that input to match to cached motionvectors or invalidators. Based on that comparison, the client thenapplies the matched motion vectors or invalidators to effect motioncompensation in a graphic interface.

It is another object of the invention to provide systems and methods forreducing input-feedback latency by caching motion vectors in a look-uptable.

It is yet another object of the invention to provide systems and methodsfor reducing input-feedback latency by associating invalidators with oneor more cached motion vectors.

It is yet another object of the invention to provide systems and methodsfor reducing input-feedback latency by instructing the client to deleteone or more cached motion vectors if the input is matched to a cachedinvalidator associated with the one or more motion vectors.

It is another objective of the present invention to disclose systems andmethods for reducing latency by caching repetitive motion vectors at aserver that transmits a previously generated motion vector library to aclient. The client stores the motion vector library, and monitors foruser input data. The server instructs the client to calculate a motionestimate from the input data and instructs the client to update thestored motion vector library based on the input data, so that the clientapplies the stored motion vector library to initiate motion in a graphicinterface prior to receiving actual motion vector data from the server.

It is another object of the present invention to disclose systems andmethods for reducing latency by caching repetitive motion vectors inwhich the server transmits a context update to the client to disableapplication of the stored motion vector library.

It is yet another object of the present invention to disclose systemsand methods for reducing latency by caching repetitive motion vectors inwhich one or more scaling factors is applied to the motion vectorlibrary.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the invention and many of the attendantadvantages thereof will be readily obtained as the same becomes betterunderstood by reference to the following detailed description whenconsidered in connection with the accompanying drawings, wherein:

FIG. 1 is a block diagram exemplarily illustrating a remote gamingsystem in which a videogame is running on a server while beingcontrolled by input at a remote client;

FIG. 2 is a flow chart exemplarily describing the reduction ininput-feedback latency in game streaming applications by applying motioncompensation for an appropriate player input;

FIG. 3 is a block diagram exemplarily illustrating an example momentduring the runtime of a videogame streaming environment that isutilizing player input motion compensation;

FIG. 4A is a diagram exemplarily illustrating an example macroblockduring the player motion compensation of FIG. 3 in which two sets oftagged motion vectors are shown;

FIG. 4B is a diagram exemplarily illustrating an example macroblockduring the player motion compensation of FIG. 3 in which summed motionvectors are shown;

FIG. 4C show a diagram exemplarily illustrating an example macroblockduring the player input motion compensation of FIG. 3 in which anexample shifted macroblock is shown;

FIG. 5 is diagram exemplarily illustrating an alternative method toapply motion vectors during player input motion compensation on theclient;

FIG. 6A shows an exemplary macroblock undergoing player input motioncompensation and blending for the alternative method shown in FIG. 5,where the example macroblock is being shifted to the right;

FIG. 6B shows an exemplary macroblock undergoing player input motioncompensation and blending for the alternative method shown in FIG. 5,where actual motion vectors are shown;

FIG. 6C show an exemplary macroblock undergoing player input motioncompensation and blending for the alternative method shown in FIG. 5,where the resulting motion is shown;

FIG. 7 is a diagram illustrating runtime generation of anticipatedmotion vectors;

FIG. 8 is a flow diagram exemplarily illustrating the transmission andclient-side storage of anticipated motion vectors for the purpose ofplayer input motion compensation;

FIG. 9 is a diagram illustrating an exemplary process of invalidatinganticipated motion vectors;

FIG. 10 is a diagram showing an exemplary method for generating themotion vector library and an example repetitive motion vector libraryfor caching purposes;

FIG. 11 is a flow diagram exemplarily illustrating the process to cache,apply, and update motion vector libraries for player input motioncompensation;

FIG. 12 is a diagram exemplarily illustrating how the motion vectormapping may be updated; and

FIG. 13 is a diagram illustrating an exemplary modification for applyingmulti-frame motion vectors during player input motion compensation,especially in the case of cached motion vectors.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In describing the preferred embodiments of the invention illustrated inthe drawings, specific terminology will be resorted to for the sake ofclarity. However, the invention is not intended to be limited to thespecific terms so selected, and it is to be understood that eachspecific term includes all technical equivalents that operate in asimilar manner to accomplish a similar purpose. Several preferredembodiments of the invention are described for illustrative purposes, itbeing understood that the invention may be embodied in other forms notspecifically shown in the drawings.

Certain types of player input make better candidates for player inputmotion compensation. Two factors contribute to a given input'ssuitability: the player sensitivity to input-feedback latency, and thedifficulty of implementing player input motion compensation withoutintroducing significant artifacting. Each input will need to beevaluated for suitability. For example, in a first-person shooter game,the player will be very sensitive to mouse-based view rotation, afraction of a second delay, as little as 16 ms, between player input andvideo output will be noticeable. However, in the same situation, gamepadcontroller-based view rotation is typically slower and players might beless sensitive to input-feedback latency. View rotation can beapproximated by shifting the scene in the opposite direction of therotation but there may be undesirable artifacting along the edge of theimage in the rotation direction. For small view rotations, such asadjusting aim on an onscreen enemy, players may not even notice edgeartifacting. In another example, accelerating a car in a driving gamemay be low priority for player input motion compensation due to lack ofplayer sensitivity and/or inertia to latency, but steering and brakinginputs may be high priority because players will notice input-feedbacklatency.

The time between receiving a player input and displaying a motion outputis the player-feedback latency. By using motion compensation, anestimated motion can provide feedback almost immediately while waitingfor the server to process the player input. In this way, theplayer-feedback latency is dramatically reduced in game streamingapplications. By implementing player input motion compensation in a gamestreaming application, a motion estimate can be provided in the nextavailable frame. By contrast, it takes several frames for input totravel to the server, produce an output frame, and return. Player inputmotion compensation may also provide some benefit in traditionalnon-streamed game applications where the game engine and renderer mayhave a player-feedback latency of a few frames.

The client will not have the appropriate context to track an object'smovement around the screen. Player input motion compensation will not besuitable for cases where the location of specific macroblocks or videoobjects is unknowable to the client. For example, on a 2D-platformergame, the character can move around the screen from left to right. Theclient will not know where the character is located when the playerpresses the jump input; therefore player input motion compensation alonecannot be used in this case to reduce input-feedback latency.

In general, the motion vectors for player input motion compensationshould be generated ahead of time. For motion such as player camerarotation, the motion vectors can be calculated based on how the gameweights the input. In certain embodiments, the motion vectors may be theinput value multiplied by the sensitivity weight. For motion that can'tbe directly calculated, such as animated movement, the animation may betriggered during development such that the motion vectors may bemeasured directly and stored. Measuring motion vectors can beaccomplished through the same motion estimation techniques performedduring H.264 encoding.

FIG. 1 illustrates an example system in which a video game is controlledby a remote client. In this system, a server 100 hosts video gamesoftware 102 and a graphics engine 104 which renders video output. Thevideo is encoded in a codec (also referred to as a coding engine orencoder) 106 and the encoded video data 108 is transmitted to a remoteclient. The server architecture 100 may be any combination of hardwareor software which can support the functions of both graphics engine 104and codec 106. In the given example, the graphics engine 104 may beimplemented as, for example, a GPU 110 controlled by video game software102 loaded into some computer readable memory 112, while the codec 106may be implemented as a CPU 114 running video encoding software.

The remote client is composed of a computer system 116 capable ofrunning a client-side codec 118 to decode the transmitted encoded videodata 108 and a client application 120 to apply the player input motioncompensation. The client computer system 116 also contains a displaycontroller 122 to drive display hardware 124. The input from theclient-side input peripherals 126 will be converted by the clientapplication 120 into control data 128 which is transmitted back to thegame software 102 running on the server 100. The input from theperipherals 126 will also be used to determine what, if any, playerinput motion compensation to apply as illustrated in more detail by FIG.2.

FIG. 2 is a flow chart describing the steps required to perform playerinput motion compensation for a single input. When the clientinitializes, the server sends a lookup table containing inputs and theirassociated motion vectors at step 200, which is then cached by theclient at step 202. In this implementation, the client is generalized toserve the needs of multiple streaming games. In certain embodiments, agame-specific client can skip steps 200 and 202 because it alreadycontains the lookup table for the game. In an alternate implementation,a game-specific client may permanently store the motion vector lookuptable without the need to cache from the server.

When the client receives player input from an input device such as amouse or gamepad controller at step 204, the client application willcheck the cached motion vector lookup table for matching inputs at step206. If there is no matching player input, the client will take noadditional action and send input to the server without additionalmodification. If there is a matching player input in the cache, theclient will apply player input motion compensation. Optionally, thecache may be capable of changing entries in the lookup table based onplayer input. For example, when the player hits the pause button, allplayer-movement inputs should be disabled until the player exits thepause screen. In one implementation, the client-controlled lookup tablemay have two sets of inputs, one for use in the pause menu and one foruse outside the pause menu, that are switched, preferably by the client,whenever the player selects the pause input. In an alternateimplementation, the server may switch the contents of the cached lookuptable on the client.

When the client application receives player input for motioncompensation, the client will add a tag to the player input and itsassociated motion vectors at step 208. The tagged input is sent to theserver at step 210. The tag is any identifier that can correlate playerinput to a future frame. For example, the tag may be an integer that isincremented each time the client receives input that will be used toperform player input motion compensation. The tag can be added asmetadata in the same network packet as the player input or sent in asimilar messaging pattern that keeps the tag information synced with theinput information. The client confirms whether or not tagged a tag isreceived at step 213. While the player input is being tagged and sent,the client applies the motion vectors contained in the cached lookuptable at step 212. These motion vectors will be applied for eachincoming frame until the correlating tag is returned from the server. Adetailed description of an exemplary method to apply these motionvectors is illustrated in FIG. 3.

When the server receives tagged player input, the tagged player input ispassed to the game, which generates an output frame at step 214. Thevideo image is then encoded at step 216. Before the encoded frame issent back to the client, the player input tag is attached to the encodedframe at step 218. This is the same tag that was previously sent withthe player input and signifies that the output frame contains the actualvideo feedback from the player input. Attaching a tag to the encodedframe can be accomplished by adding the tag as metadata to the samenetwork packet as the encoded frame. The tagged encoded frame is sentback to the client at step 220. When the client receives an encodedframe with a tag, the client can correlate the tag to a previous playerinput motion compensation. The client then stops applying the previousmotion compensation at step 222.

FIG. 3 is an illustration of an example moment during the runtime of avideogame streaming environment that is utilizing player input motioncompensation. When the client receives any player input at step 300, itwill be compared to the motion vector lookup table at step 302. If thereis a matching player input, the associated motion vectors will be usedfor player input motion compensation. The motion vectors are tagged atstep 306 with a unique input tag at step 304. In this example, thechosen tag is the integer “1003.” The tagged motion vectors are added toa queue at step 308 containing any other tagged motion vectors currentlybeing used for player input motion compensation.

The next frame arrives in the bitstream from the server at step 320.This frame is tagged with a unique identifier at step 318, in this case,the integer “1001,” which indicates that the frame contains theresultant motion of all previous player inputs up to and including theinput corresponding to tag “1001.” The tag “1001” indicates to theclient that it can stop applying motion compensation at step 322 forthis tagged input. The motion vectors with the tag “1001” are thenremoved from the tagged motion vector queue at step 308 along with anymotion vectors with earlier tags that may remain in the queue in thecase that previous packets have been lost.

The coded video 316 is decoded at step 324. Meanwhile, the remainingmotion vectors in the motion vector queue at step 308 are summed at step310. The motion vectors are typically vector fields with a vector foreach macroblock in the image. To sum the motion vectors, the vectors aresummed element-wise so that the result is a vector field with a vectorfor each macroblock. The sum of two vector fields is the vector sum ofthe vectors for each point in the field, so the sum of two sets ofmotion vectors is the vector sum for each macroblock in the image. Thesum of two vectors is defined as the component-wise sum of theircomponents, which may be represented as {u₁, u₂}+{v₁, v₂}={u₁+v₁,u₂+v₂}. In this example, two sets of motion vectors with tags “1002” and“1003” are contained in the queue; these two sets of motion vectors aresummed. The tags are chronological in nature, which allows the client toknow the ordering of the previously tagged player inputs. This lets theclient discard tagged motion vectors up to and including the return tagin an incoming frame. Additionally, the tagging discussed above iscomputationally cheaper than more complex methods. Optional smoothingfunctions can be applied at this point to prevent clamping artifacts orto mitigate similar introduced-artifacting.

Clamping artifacts are introduced when macroblocks are moved offscreenand manifest as a smearing of pixel color perpendicular to the screenedge. An example smoothing function would reduce the strength ofoutward-pointing motion vectors as they get closer to the edge of theimage. Only the outward-directional component needs to be weakened, they-component for outward-pointing vectors towards the top and bottomedges of the image and the x-component for outward-pointing vectorstowards the left and right edges of the image. For vectors that pointtowards the border, the vector component could be multiplied by thesquare of the distance from the edge so that as the distance from theborder approaches zero, the vector component will approach zero. In thisexample, an outward-pointing vector towards the right edge of the imagewill be transformed from {x,y} to {x*d*d,y} where d is the distance fromthe edge. This will mitigate the clamping artifacts in exchange for someslight image distortion around the border of the image. The distortionis far less noticeable to the player than the clamping artifacts.

After the decoding process completes at step 324 on the encoded videoframe, the summed motion vectors are used in motion compensation at step312. The resulting video is output at step 314. This output contains theplayer input motion compensation data and will be displayed on theclient. This outputted frame is the first frame containing a motionestimate for the input with correlation tag “1003.” This outputted framealso contains a motion estimate for the previous input with correlationtag “1002,” for which the client is still waiting for the server toreturn actual motion vectors. This outputted frame is also the firstframe to contain the actual motion vectors for a previous input withcorrelation tag “1001,” for which the client had been previouslyestimating motion. As a consequence, three motion estimation statesexist at this point in the method: one new estimation state for newinputs; one continued estimation state that is awaiting actual results;and another estimation state that is stopped because actual results havearrived at the client.

FIG. 4 is a diagram illustrating an example macroblock during the playerinput motion compensation step of FIG. 3. The player input motioncompensation step is procedurally similar to the process performedduring H.264 decoding where decoded motion vectors are used to determinewhere macroblocks in the current frame have moved from in the previousframe. For player input motion compensation, the applied motion vectorswill estimate the future feedback for a player input by movingmacroblocks in the current frame before displaying the output video tothe player. FIG. 4A shows the two sets of tagged motion vectors in thequeue at step 400 from the example tagged motion vectors shown at step308 in FIG. 3. These two sets are summed by taking the vector sum foreach macroblock in the image to create one set of motion vectors at step402 in FIG. 4B. Each one of the motion vectors in this set representsthe estimated movement for each macroblock in the current frame. Foreach vector in the summed motion vectors at step 402, a correspondingmacroblock in the example image of FIG. 4C will be shifted. One examplemacroblock is shown in FIG. 4C being shifted by its corresponding playerinput motion vector at step 404. Each macroblock in the example imagewill be shifted in this manner. The example image in FIG. 4C containsonly 50 macroblocks, but a high definition image will contain thousandsof macroblocks. The example motion vectors shown in FIG. 4 show uniformrigid motion, but the described player input motion compensationtechnique can be used with motion vectors of arbitrary complexity todescribe rotations, vibrations, or other complex motions in screenspacethat are known in the art.

FIG. 5 is an illustration of an alternative method to apply motionvectors during player input motion compensation on the client. Incertain embodiments of this method, it is not necessary to handleresiduals during encoding and decoding of video. After a matching playerinput is found in the lookup table, the associated motion vectors aretagged at step 500 and used for player input motion compensation asshown in steps 208 and 212 in FIG. 2. During the decoding process on thenext frame, the player input motion vectors are added to the decodedmotion vectors and injected into the motion compensation step at step502, defined in the H.264 coding standard. The deblocking filter, asdefined by the H.264 coding standard, is applied at step 504, but shouldonly be applied to blocks that have not been modified by player inputmotion compensation. The resulting output, shown at step 506, containsthe player input motion compensation and will be displayed on theclient. The output is subsequently buffered at step 506 and at step 518,becomes the previous image for use in motion compensation on the nextframe. Unlike the implementation in FIG. 3, in this implementation, thetagged player input motion vectors are only applied once. This meansthat they do not need to be summed with all the tagged motion vectors inthe queue, because the previous image, shown at step 518, will containthe sum of previous tagged motion vectors. The time between receivingthe input and displaying the video output is the input-feedback latency,which will have been significantly reduced by using player input motioncompensation instead of waiting for the server to return output video.

At the same time as step 500, the client will send the correspondingtagged player input to the server, as shown in steps 208 and 210 in FIG.2. The client will eventually receive coded video as shown at step 508with the same tag 510 in the bitstream from the server at step 512, asshown in step 220 in FIG. 2. The tag 510 associated with the videosignifies that the actual motion previously estimated by player inputmotion compensation is represented in the encoded video, as shown atstep 508. The encoded video passes through entropy decoding at step 514and inverse quantization & inverse transformation at step 516, asdefined by the H.264 coding standard.

The previous player input motion compensation step, which has beenperformed prior to steps 508, 510, 512, 514, and 516, described in thepreceding paragraphs, has already shifted the macroblocks that theactual motion vectors from the encoded frame are trying to shift.Therefore, applying the actual motion vectors directly will shiftincorrect macroblocks. The bitstream will contain two related pieces ofdata: the encoded video frame and a correlation tag. The encoded frameis sent to entropy decoding at step 514, while the tag is sent ahead tothe blending process at step 520. To compensate, the blending process atstep 520 supplies correction motion vectors by calculating thedifference between the incoming actual motion vectors and the previouslyused player input motion vectors 500 with a matching correlation tag.The difference can be calculated by adding the inverse motion vectorsfor the correlation tag to the actual motion vectors and is described inmore detail in connection with FIG. 6. The correction vectors are usedin place of the decoded motion vectors for motion compensation at step502. The remainder of the decoding pipeline continues as usual with thedeblocking filter at step 504, the next output video frame at step 506,and buffering the output at step 518. These steps may continue to repeatfor each frame indefinitely. In general, the blending process of step520 occurs after the inverse quantization (step 514) and inversetransform (step 516), since the actual motion vectors will not beavailable until this point. Once the actual motion vectors areavailable, the blending process will use them to calculate thedifference between actual and estimated motion vectors.

FIG. 6 illustrates an example macroblock undergoing player input motioncompensation and blending. At time 0 ms, the player presses the input torotate the camera view to the left. The associated player input motioncompensation motion vectors, shown as “PIMC” 600, from the lookup tableare applied to all macroblocks in the next frame. FIG. 6A shows anexample macroblock being shifted to the right. The results of playerinput motion compensation appear in the next decoded frame, resulting ina maximum input-feedback latency of 16 ms, the length of one frame, forvideo running at 60 frames per second. In other embodiments, the maximuminput-feedback latency may be 33 ms for video running at 30 frames persecond. It is thus contemplated that the invention operates across avariety of framerates, limiting the maximum input-feedback latency tothe length of one frame. When the tagged video returns from the server,it contains the actual motion vectors 602 encoded by the server, asshown in FIG. 6B. For this example, the coded video returns at time 100ms but this will be heavily dependent on network latency. The actualvectors 602 refer to macroblocks that have already been shifted duringplayer input motion compensation 600, so the actual vectors 602 cannotbe applied directly to the existing frame. Instead, the correctionvectors 604 need to be calculated by finding the difference between theactual vectors 602 and the player input motion vectors 600. Thedifference can be calculated by adding the inverse player input motionvectors to the actual motion vectors. Finding these vector differencesis referred to as blending at step 524 in FIG. 5. The smaller thecorrection vector 604, the more successful the player input motioncompensation method was at estimating the actual motion vectors. Theresulting motion 608 shown in FIG. 6C for the example macroblock is thesame as the actual motion vector 602. This example illustrates howplayer input motion compensation can estimate the video feedback for aplayer input and show the result for the time between 16 ms and 100 ms,while an unmodified system would not display any player-feedback until116 ms after the player input is received.

During development, game developers will need to decide which motionsand animations will send anticipated motion vectors during runtime.Unique yet predictable motion vectors are the best candidates foranticipated motion vectors. A categorical example would includeanimations that are adaptively altered by the engine, such as animationsthat use kinematics equations to calculate joint angles, animations thatare time-warped, or animations that are otherwise stretched orcompressed. For example, a ledge grab animation plays when a player getswithin range of a defined ledge and jumps. The ledge grab animation isstretched so that the player's hands are on the ledge but are stillattached to the player's body. The animation plays out over a definednumber of frames, placing the player on top of the ledge. The startpoint in this example is variable, with a range of acceptable locationsand orientations. This ledge grab animation is a good candidate forgenerating anticipated motion vectors because the exact animation is notknown ahead of time but is programmatically generated by the game engineon-demand. The client especially cannot know the motion vectors for thisanimation as the client does not have any contextual information aboutthe player location in a game streaming environment.

Anticipated motion vectors will only be useful over a limited contextualor temporal scope such as a specific camera location, a small window intime, or some other specific player-context. For each set of anticipatedmotion vectors, a corresponding invalidator needs to be generated. Theinvalidator may be used on the client to prevent anticipated motionvectors from being applied after they would be valid. In certainembodiments, an invalidator may be a set of any player inputs that wouldchange the game context, such that playing the anticipated motionvectors is no longer viable. In other embodiments, an invalidator may bea time window for which anticipated motion vectors may be valid. In yetother embodiments, an invalidator may be a combination of invalidatinginputs and a time window. For example, the anticipated motion vectorsgenerated for a ledge grab animation are only valid within a limitedplayer location and orientation, and as such, an invalidator wouldnecessarily include any translational or rotational movement input.Invalidators will need to be designed and implemented during thedevelopment of the anticipated motion vector feature. Anticipated motionvectors may also be disabled or updated by events or messages sent fromthe server as described below in connection with FIGS. 10 through 13,which discuss using cached repetitive motion vectors for motioncompensation.

Anticipated motion vectors may be generated ahead of time or they may begenerated as needed during runtime. For animations that have a limitednumber of permutations, for instance, the anticipated motion vectors canbe generated offline by triggering each permutation and recording themotion vectors. In certain embodiments, for a generalized client, themotion vectors would be stored server-side and then sent to the clientto be cached on-demand. When the motion vectors are sent to the client,they are cached in the lookup table. Pre-generated anticipated motionvectors may be stored at the server and will be available as agame-readable file format which allows the server to send pre-generatedmotion vectors to be cached in the lookup table on the client during theruntime of the game. Animations that are generated during runtime, suchas animations calculated through inverse kinematics, cannot bepre-generated because there may not be a discrete number of possibleanimation permutations. Inverse kinematics is a method commonly used inreal-time rendering to fit an animation within a set of boundaryconditions. For example, a player character in a videogame wants to graba nearby ledge, the boundary conditions will be defined by the locationswhere the player's hands hit the ledge and the ledge-grab animation willbe altered accordingly through inverse kinematics. For adaptivelyaltered animations such as these, the game may speculatively renderpossible animations into an offscreen motion vector image during runtimeand record the anticipated motion vectors as needed. For example, if theplayer is near a grabbable ledge, the game may anticipate the playerwill grab the ledge soon, and the game may speculatively render theledge grab animation to generate anticipated motion vectors. Adaptivelyaltered animations that will need anticipated motion vectors to begenerated during runtime will need to be identified by a developer aheadof time.

Existing game systems that describe player context, such as playerlocation tracking, scripting systems, trigger volumes, or pathfindingsystems, can be used to generate an event that will signal when the gameneeds to speculatively render an animation. For example, a game maytrack the player's proximity to a grabbable-ledge and signal the game tospeculatively render the ledge grab animation and record the anticipatedmotion vectors. Certain animations like picking up a weapon, pulling alever, or pushing a button may be stretched or adjusted based on playerproximity to the interaction and player orientation. These animationshave too many permutations to make pre-generation feasible, but couldalso be generated during runtime, as exemplarily shown in FIG. 7.Motions that play out in the same manner every time can be generated andrecorded offline. These are typically motions that occur in the samescreen space at the same rate every time they are triggered. Motionvectors for these animations can be recorded offline by triggering allpossible permutations of the animation and recording the game-generatedmotion vectors or generating motion vectors through more traditionalmotion estimation techniques such as those used in the H.264 codec. In apreferred embodiment, game-generated motion vectors, as described abovein connection with FIGS. 1 through 6, are used to ensure high qualitymotion estimates. This process can occur at any point duringdevelopment, but it is preferable to add this process as a stage duringthe build process or some other existing asset-conditioning process,such as pre-generating mip-maps and the level of details (“LODs”). Assetconditioning may include any process that compiles game assets fromhuman-readable source formats into machine-readable formats. Forexample, mip-maps may be generated ahead of time by convertingartist-created texture files into game-ready formats that containmultiple resolutions. Similarly, LODs may be generated ahead of time byconverting artist-created model files into game-ready formats thatcontain multiple levels of detail. Motion vector generation can be addedto an existing asset-conditioning process that converts artist-generatedanimation formats into game-ready formats.

FIG. 7 illustrates an example method for generating motion vectors inoffline or runtime scenarios. The animations can be rendered to anoff-screen surface/image, step 700, and the motion vectors can berecorded for immediate use. Only the portions of the screen that movewill need to be rendered, the other objects in the scene can be ignored.The dashed object, shown at step 702, and solid object, shown at step704, represent the position of an animated object in a previous frameand current frame respectively. The movement from the previous frame tothe current frame will be captured at step 706 in the form of motionvectors, shown at step 708. The motion vectors may be captured from thegame-generated motion vectors or captured through more traditionalmotion estimation techniques such as those used in the H.264 codec. In apreferred embodiment, game-generated motion vectors, as described inabove in connection with FIGS. 1 through 6, are used to ensure highquality motion vectors. Several frames' worth of anticipated motionvectors can be quickly calculated for a given animation by repeating theprocess illustrated by steps 700 through 708, until motion vectors havebeen generated for all required frames. Not all of the frames in theanimation need to be generated, just enough need to be generated to playon the client while the video stream catches up. The minimum number ofgenerated frames will depend on the delay between sending player inputand receiving the resultant video stream on the client; and the lengthof the generated portion of an animation should be at least as long asthe delay. The anticipated motion vector frames may be rate-scaledduring playback as described below in connection with FIGS. 10 through13, which discuss the use of cached repetitive motion vectors in motionestimation. If the playback of anticipated motion vector frames israte-scaled, generating anticipated motion vectors for a portion of ananimation equal to the delay will result in a playback rate scaling of50%. Generating motion vectors for a longer portion of the animationwill result in playback that is less-aggressively rate-scaled.

The macroblock size used for video encoding should be considered whenrecoding the motion vectors, and there should be a motion vector foreach macroblock. In the preferred embodiment, game-generated motionvectors are generated as per-pixel motion vectors and transformed intoper-macroblock motion vectors by finding the arithmetic mean for eachmacroblock group of per-pixel motion vectors.

FIG. 8 illustrates the transmission and client-side storage ofanticipated motion vectors for the purpose of player input motioncompensation. During the development of the video game software, eventsneed to be configured to signal upcoming input-driven context-sensitiveanimations. For example, a game developer would like to send anticipatedmotion vectors so that they will be available when the player performs aledge grab animation. The developer implements an event that will betriggered whenever the player is both facing and within range of agrabbable-ledge. In this example, when the player gets close to agrabbable-ledge while playing the game, the example event is received at“DURING GAME RUNTIME, RECEIVE AN EVENT,” step 800. The event type willdescribe whether the anticipated motion vectors will need to begenerated, as in the case of adaptively altered animations, or whetherthe anticipated motion vectors have been pre-generated offline, as inthe case of animations which are never altered but play infrequently orare dependent on player-context. In the above example, the ledge grabanimation is stretched based on the player's distance from the ledge,which means that motion vectors will need to be generated at runtime at“GENERATE ANTICIPATED MOTION VECTORS,” step 802. In another case, themotion vectors may have been generated offline and read from storage at“READ PRE-GENERATED MOTION VECTORS,” step 804.

One example of pre-generated motion vectors may be switching betweenweapons in a large arsenal. The number of possible weapon-switchpermutations can grow quite large, making it impractical to cache theentire set of resulting motion vectors. In general, if motion vectorstake up an excessive quantity of space in the limited cache and are notused frequently enough, they are not prime candidates for pre-caching.The anticipated motion vectors are sent to the client at “SENDANTICIPATED MOTION VECTORS AND INVALIDATORS,” step 806. The anticipatedmotion vectors are added to the motion vector lookup table at “CACHEANTICIPATED MOTION VECTORS AND INVALIDATORS,” step 808. In oneembodiment, the invalidation system functions similarly to the systemthat triggers the application of motion vectors in the lookup table butdisables the motion vectors instead. When a set of motion vectors andinvalidators is received at “CACHE ANTICIPATED MOTION VECTORS ANDINVALIDATORS,” step 808, the invalidators will need to be registered bythe invalidation system.

The player input motion compensation method, as described above inconnection with FIGS. 1 through 6, compares all player inputs to theentries in the motion vector lookup table. In the above example, whenthe player enters the required input to initiate the ledge grabanimation, the input will match the previously cached input in thelookup table at “MATCHING PLAYER INPUT RECEIVED,” step 810. If thecached anticipated motion vectors have not yet been invalidated, theanticipated motion vectors will be applied at “APPLY PLAYER INPUT MOTIONCOMPENSATION,” step 812. If a matching player input will invalidate theanticipated motion vectors, or if the anticipated motion vectors expireafter a predetermined time, or if the anticipated motion vectors expireafter being applied once, the anticipated motion vectors are removedfrom the lookup table at “INVALIDATE,” step 814 and are not applied.

FIG. 9 describes an exemplary method of signals that may invalidate ananticipated motion vector set. Invalidation is a type of update to thelookup table where a set of anticipated motion vectors is removed fromthe lookup table, which is preferably cached at the client. In additionto responding to update events received from the server, the updatemechanism may monitor for player inputs and contain invalidationcountdown timers. When a set of anticipated motion vectors is cached inthe lookup table, its invalidator will likely be registered with theupdate function/mechanism. An invalidator is any data signal thattriggers an invalidation update to the lookup table. The example lookuptable, “Lookup Table,” shown at step 900, contains three sets ofanticipated motion vectors: an anticipated door animation, shown at“ANTICIPATED DOOR ANIMATION,” step 902, an anticipated ledge grabanimation, shown at “ANTICIPATED LEDGE GRAB,” step 904, and ananticipated kill shot animation, shown at “ANTICIPATED KILL SHOTANIMATION,” step 906. Anticipated motion vectors may be invalidatedafter being applied once. In the example, the anticipated door animationmotion vectors shown at “ANTICIPATED DOOR ANIMATION,” step 902, areapplied when the player presses the input at “OPEN DOOR INPUT,” step908, to open a nearby door. Simultaneously, the open door input at “OPENDOOR INPUT,” step 908, is deregistered and the anticipated dooranimations, shown at “ANTICIPATED DOOR ANIMATION,” step 902, may beremoved from the lookup table at “INVALIDATE AFTER USE,” step 910.Similarly, other inputs may invalidate anticipated motion vectors beforethey are applied. In the example, a set of anticipated motion vectorsfor a ledge grab animation shown at “ANTICIPATED LEDGE GRAB,” step 904,are only valid within a limited distance from a ledge.

If the player moves away from the ledge by using the movement input,shown at “MOVEMENT INPUT,” step 912, before the player jumps to grab theledge (by using the jump input, shown at “JUMP INPUT,” step 914), thenthe anticipated ledge grab motion vectors shown at “ANTICIPATED LEDGEGRAB,” step 904 will be invalidated at “INVALIDATE ON INPUTS,” step 916.Other examples of invalidating inputs may be cases in which the playerhas a grappling hook or some other movement-based weapon that wouldinvalidate anticipated motion vectors, and cases in which the playerpushes a button that activates a special ability. As invalidating inputsare context-specific, others may be apparent based on the particularimplementation. Anticipated motion vectors may also expire over time. Inthe example, a kill shot opportunity is only presented to the player fora three-second window. If the player does not press the melee input,shown at “MELEE INPUT,” step 918, within the three-second window, themotion vectors for an anticipated kill shot animation, shown at“ANTICIPATED KILL SHOT ANIMATION,” step 906, will be invalidated at“INVALIDATE ON EXPIRATION TIMER,” step 920. Anticipated motion vectorsmay have multiple invalidators and vice versa. For example, ananticipated ledge grab may be invalidated if a movement input isreceived or after the ledge grab motion vectors are used, whichevercomes first.

FIG. 10 shows an example method for generating the motion vector libraryand an example repetitive motion vector library for caching purposes.Since the motions selected are highly repetitive in nature, they willplay in the same manner every time they are triggered. This makes itpossible to generate motion vectors ahead of time and organize them intolibraries. Motion vector library generation can occur at any pointduring development, but it may make sense to add this process as a stageduring the build process or some other asset conditioning phase. In thisexample, a motion vector library will be generated for each availableweapon in a first-person shooter. When library generation for the firstweapon begins at “LIBRARY GENERATION,” step 1000, the first weaponanimation is triggered at “TRIGGER ANIMATION,” step 1002 and the motionvectors are recorded at “RECORD MOTION VECTORS,” step 1004. The motionvectors may be game-generated or generated through more traditionalmotion estimation techniques such as those used in the H.264 codec. In apreferred embodiment, game-generated motion vectors, as described inU.S. Provisional Application Nos. 62/488,256 and 62/634,464,incorporated here in their entireties, are used to ensure high qualitymotion estimates. If the recorded motion vectors are not accurate orcorrectly quantized, they will introduce artifacts when used duringplayer input motion compensation. The steps 1002 and 1004 repeat untilthe motion vectors have been recorded for each highly-repetitiveanimation in the library. Library generation starts again at step 1000until all libraries are generated.

The example shown at “MOTION VECTOR LIBRARY,” step 1006 is a greatlysimplified version of a repetitive motion vector library for a plasmarifle weapon in a first-person shooter game. This example is simplifiedto include two simple animations: one 2-frame animation for the feedbackanimation that plays when the player fires the plasma rifle, shown atstep 1008, and one 4-frame animation for the bobbing motion of the riflethat occurs when the player walks forward, shown at step 1010. In atypical, real-world environment, a weapon would likely have moreanimations and the animations may be much longer than four frames.

FIG. 11 illustrates the process to cache, apply, and update motionvector libraries for player input motion compensation. When the gameinitializes, the server will send the pre-generated motion vectorlibraries to the client at “ON STARTUP, SEND REPETITIVE MOTION VECTORLIBRARIES,” step 1100. The client stores the motion vector libraries inmemory, typically in the form of a lookup table, at “CACHE REPETITIVEMOTION VECTORS,” step 1102. In this implementation, the client isgeneralized to serve the needs of multiple streaming games. In analternate implementation, a game-specific client may permanently storethe motion vector libraries without the need to cache repetitive motionvectors from the server.

At this point, the client can start monitoring for player input andcomparing the incoming inputs to entries in the player input motioncompensation lookup table. When an incoming input has a matching entryin the lookup table at “MATCHING PLAYER INPUT RECEIVED,” step 1104, themotion vectors associated with the player input are used to provide amotion estimate as exemplarily described in connection with FIGS. 1through 10, at “APPLY PLAYER INPUT MOTION COMPENSATION,” step 1106.There may be certain cases where receiving a specific input may requirethe lookup table to be altered at “UPDATE CACHED REPETITIVE MOTIONVECTOR MAPPING,” step 1112. An example of an input that would alter thelookup table is the pause button, which may disable player inputs suchas character movement, weapon firing animations, or other play-relatedmotions. After applying the motion vectors and optionally updating thelookup table, the client will continue to monitor for player input.After the lookup table is updated at “UPDATE CACHED REPETITIVE MOTIONVECTOR MAPPING,” step 1112, incoming player input will be compared tothe updated entries in the lookup table.

At any point during the game's runtime, a change in context for theplayer may require the lookup table to be altered. For example, theplayer may switch weapons requiring the cached motion library to beswitched from the previously held weapon to the new weapon. An exemplaryimplementation may see the server-side game monitoring for specific gameevents. These events will be configured by methods known in the artduring game development and may be sent over the game's existingmessaging or event system. When a running game instance receives one ofthese events at “DURING RUNTIME, RECEIVE AN EVENT,” step 1108, a messagewill be generated and transmitted to the client at “SEND CONTEXTUPDATE,” step 1110. A context update may be sent for any game event thatchanges the relationship between player input and the motion vectorscontained in the lookup table. Context updates may enable or disable aplayer input from triggering a set of motion vectors, change the motionvectors associated with a given input, or otherwise add or removeassociations between player input and motion vectors for player inputmotion compensation. When the message reaches the client, the lookuptable is altered according to the changing context at “UPDATE CACHEDREPETITIVE MOTION VECTOR MAPPING,” step 1112.

FIG. 12 is a diagram illustrating how the motion vector mapping may beupdated. The lookup table 1200, which was stored in the client's cache1202, is used by the client to find which motion vectors are associatedwith a given player input. There may be times during runtime of the gamewhere contextual changes alter the behavior of player inputs. Forexample, if a player is moving forward but hits a movement blocker suchas a wall, the predicted motion vectors for forward movement will needto stop being applied. When this happens, the server will send a contextupdate to the client as shown in step 1110 in FIG. 11. This movementblocking event 1204 is received, which signals the client to disable theconnection between the move forward input 1206 and the correspondingmotion vectors 1208. The forward movement motion vectors will stop beingapplied when the player uses the forward movement input until anotherevent from the server re-enables the connection between the move forwardinput and the move forward motion vectors. For this example, the gamewill generate an event when the player's movement becomes unblocked andtransmit the context update to the client to reestablish the connectionbetween the forward input and the move forward motion vectors in thelookup table.

In another example, the player is holding the plasma rifle but switchesto a shotgun. The plasma rifle motion vector library 1210 is stored inthe cache along with its firing motion vectors 1209, which correspond toa specific firing input 1211. The cache also stores motion vectorlibraries for other weapons including a shotgun 1212 and pistol 1214.When the client receives the weapon switch event 1216 from the server,the plasma rifle motion vector library 1210 is switched out of thelookup table 1200 for the shotgun motion vector library 1212. To preventplayer input motion compensation from improperly occurring during theweapon switch, two events can be used in tandem to first disable theplasma rifle motion vector library 1210 while a weapon switch animationplays, and then switch the two motion vector libraries after the weaponswitch animation completes.

For longer multi-frame motion vectors, it is possible to stretch outtheir application such that the last frame of motion vectors is appliedas the last frame of actual video is received from the server. This willallow the server to catch up to the estimated motion at the moment whenthe client runs out of cached motion vectors. The scaling factor for themotion vectors is defined as the playbackSpeedScale, exemplarilycalculated as shown below.

${playbackSpeedScale} = \frac{{cached}\mspace{14mu} {animation}\mspace{14mu} {time}}{{{cached}\mspace{14mu} {animation}\mspace{14mu} {time}} + {delay}}$

Where the delay is defined as the time between the initial player inputevent and receiving the actual video on the client. This delay includesthe time it takes to send the input over the network to the server,processing on the server including game logic, rendering logic, GPUrendering time, and encoding time, and the network time to return thevideo back to the player. The delay should already be continuouslymeasured in any game streaming environment. The preferred embodiment ofplayer input motion compensation uses a correlation tag, as described inU.S. Prov. App. Nos. 62/488,256 and 62/634,464, to correlate playerinput and actual video. Before an incoming player input is sent to theserver, a correlation tag is attached as a unique identifier. When avideo frame returns from the server with a correlation tag, the clientmatches the unique identifier to a previous input. This signals theclient to stop estimating motion for the correlated input or undo aprevious motion estimation through blending techniques. The length ofthe cached portion of the animation, or cached animation time, can becalculated by multiplying the number of frames in the cached animationby the length of each frame.

In FIG. 13, an exemplary animation containing 10 frames is used forplayer input motion compensation in a game running at 60 frames persecond with a delay of 100 ms. When the player input triggers the playerinput motion compensation, the playbackSpeedScale is calculated for thecached motion vectors as shown below.

${playbackSpeedScale} = \frac{{cached}\mspace{14mu} {animation}\mspace{14mu} {time}}{{{cached}\mspace{14mu} {animation}\mspace{14mu} {time}} + {delay}}$${playbackSpeedScale} = \frac{10\mspace{14mu} {frames}*\frac{1000\mspace{14mu} {ms}}{60\mspace{14mu} {frames}}}{\left( {10\mspace{14mu} {frames}*\frac{1000\mspace{14mu} {ms}}{60\mspace{14mu} {frames}}} \right) + {100\mspace{14mu} {ms}}}$${playbackSpeedScale} = {\frac{166.66\mspace{14mu} {ms}}{{166.66\mspace{14mu} {ms}} + {100\mspace{14mu} {ms}}} = 0.625}$

The player input was received at time 0 ms. The playback rate of themotion vectors is scaled 1300 by the calculated playbackSpeedScale. Thefirst frame of motion vectors is applied to the next available frame inthe video stream from the server. The scaled motion vector frames willbe interpolated to keep the animation smooth. Since the motion vectorframes are scaled over multiple frames, interpolation is a method whichmay be used to calculate “how much” of the motion vector to apply on anygiven frame. An exemplary implementation may use a linear interpolationbased on the calculated playbackSpeedScale. For our example, thecalculated playbackSpeedScale is 0.625 which will stretch one set ofmotion vectors over 1.6 display frames. Interpolation is a method tocalculate how much of a motion vector to apply on a given frame. Thatis, interpolation calculates how far to move a macroblock down a motionvector when the set of motion vectors is stretched over multiple displayframes. Only a portion of the first scaled motion vectors should beapplied on the first display frame at 17 ms, equal to theplaybackSpeedScale of 0.625. On the second display frame at 33 ms, theremainder of the first scaled motion vectors is applied, calculated as1−0.625=0.375, then the first portion of the second scaled motionvectors is applied, calculated as the playback speed scale minus theremaining portion of the first scaled motion vectors or0.625−0.375=0.25. On the third display frame at 50 ms, the second set ofscaled motion vectors continues to be applied, the macroblocks are movedthe next 62.5% down the motion vectors. On the fourth display frame at67 ms, the remainder of the second scaled motion vectors is applied,calculated as 1−0.25−0.625=0.125, and the first portion of the thirdscaled motion vectors is applied, calculated as the playback-SpeedScaleminus the remaining portion of the second scaled motion vectors0.625−0.125=0.5. Linear interpolation continues as the scaled motionvectors are applied.

Multi-frame motion vectors may send a correlation tag for each frame ofthe cached animation to correlate each frame of estimated motion tofuture actual video.

The delay will depend greatly on the network path and architecturebetween client and server. This example uses a 100 ms delay but delayscould vary between dozens to hundreds of milliseconds. Shorter delayswill provide a better player experience, but player input motioncompensation techniques can help to disguise the impact of high delaytimes in certain cases. After a delay 1304, the actual video is received1302. For edge-located servers, or servers that are physically close toconsumers, delay times might be as low as 30 ms. For more typical serverlocations, 100 ms is more likely. The actual video retains the originalanimation length 1306 because it has not been scaled. The actual videois applied in accordance with player input motion compensationtechniques.

If the client cannot perfectly blend out previous motion compensation,the H.264 coding standard provides a redundant slice feature that cancorrect any temporally propagated errors. Based on the H.264 profilesettings, each slice will be encoded as an intra slice (I-slice) andsent on a rotating schedule at a certain frequency. Since intra sliceswill not contain motion vectors, the blend motion vectors should beapplied only when actual motion vectors arrive in p-slices. This willprevent blend motion vectors from being applied on macroblocks that haveappeared in an I-slice before the tagged frame returns from the server.

The foregoing description and drawings should be considered asillustrative only of the principles of the invention. The invention isnot intended to be limited by the preferred embodiment and may beimplemented in a variety of ways that will be clear to one of ordinaryskill in the art. Numerous applications of the invention will readilyoccur to those skilled in the art. Therefore, it is not desired to limitthe invention to the specific examples disclosed or the exactconstruction and operation shown and described. Rather, all suitablemodifications and equivalents may be resorted to, falling within thescope of the invention.

1. A computer-implemented method for caching motion vectors comprising:generating one or more motion vectors at a server, wherein the motionvectors are generated based on predetermined criteria; transmitting thegenerated motion vectors and one or more invalidators to a client,wherein the generated motion vectors and invalidators are cached at theclient; transmitting an instruction to the client to receive input froma user, wherein the client is configured to match the input to cachedmotion vectors or invalidators; and transmitting an instruction to theclient to apply the matched motion vectors or invalidators to effectmotion compensation in a graphic interface.
 2. The method of claim 1,wherein the motion vectors are cached in a look-up table.
 3. The methodof claim 1, wherein the one or more invalidators are associated with oneor more cached motion vectors.
 4. The method of claim 3, wherein the oneor more invalidators are configured to disable the one or more cachedmotion vectors.
 5. The method of claim 1, further comprising the step ofinstructing the client to delete one or more cached motion vectors ifinput is matched to a cached invalidator associated with the one or moremotion vectors.
 6. The method of claim 1, wherein the motion vectors areunique and predictable.
 7. The method of claim 1, wherein the motionvectors are generated ahead of time or during runtime.
 8. The method ofclaim 1, wherein the motion vectors are stored at the server and sent tothe client to be cached on-demand.
 9. The method of claim 1, wherein themotion vectors are game-generated.
 10. The method of claim 9, whereinthe game-generated motion vectors are generated as per-pixel motionvectors and transformed into per-macroblock motion vectors.
 11. A systemfor caching motion vectors, wherein over a network, a server: generatesone or more motion vectors, wherein the motion vectors are generatedbased on predetermined criteria; transmits the generated motion vectorsand one or more invalidators to a client, wherein the generated motionvectors and invalidators are cached at the client; transmits aninstruction to the client to receive input from a user, wherein theclient is configured to match the input to cached motion vectors orinvalidators; and transmits an instruction to the client to apply thematched motion vectors or invalidators to effect motion compensation ina graphic interface.
 12. The system of claim 11, wherein the motionvectors are cached in a look-up table.
 13. The system of claim 11,wherein the one or more invalidators are associated with one or morecached motion vectors.
 14. The system of claim 13, wherein the one ormore invalidators are configured to disable the one or more cachedmotion vectors.
 15. The system of claim 11, further comprising the stepof instructing the client to delete one or more cached motion vectors ifinput is matched to a cached invalidator associated with the one or moremotion vectors.
 16. The system of claim 11, wherein the motion vectorsare unique and predictable.
 17. The system of claim 11, wherein themotion vectors are generated ahead of time or during runtime.
 18. Thesystem of claim 11, wherein the motion vectors are stored at the serverand sent to the client to be cached on-demand.
 19. The system of claim11, wherein the motion vectors are game-generated.
 20. The system ofclaim 19, wherein the game-generated motion vectors are generated asper-pixel motion vectors and transformed into per-macroblock motionvectors.